---
title: Router Tracing
subtitle: Collect tracing information from the router
description: Collect and export tracing information from the Apollo GraphOS Router or Apollo Router Core. Supports OpenTelemetry Protocol (OTLP), Datadog, New Relic, Jaeger, Zipkin.
context:
  - telemetry
redirectFrom:
  - /graphos/routing/observability/telemetry/trace-exporters/overview
---

Apollo Router supports a collection of tracing exporters:

* [OpenTelemetry Protocol (OTLP)](/router/configuration/telemetry/exporters/tracing/otlp) over HTTP or gRPC
* [Datadog (via OTLP configuration)](/router/configuration/telemetry/exporters/tracing/datadog)
* [Dynatrace (via OTLP configuration)](/router/configuration/telemetry/exporters/tracing/dynatrace)
* [Jaeger (via OTLP configuration)](/router/configuration/telemetry/exporters/tracing/jaeger)
* [New Relic (via OTLP configuration)](/router/configuration/telemetry/exporters/tracing/new-relic)
* [Zipkin](/router/configuration/telemetry/exporters/tracing/zipkin).

The router generates [**spans**](/router/configuration/telemetry/instrumentation/spans) that include the various phases of serving a request and associated dependencies. This is useful for showing how response time is affected by:

* Sub-request response times
* Query shape (sub-request dependencies)
* Router post-processing

Span data is sent to a collector such as [Jaeger](https://www.jaegertracing.io/), which can assemble spans into a Gantt chart for analysis.

<Tip>

To get the most out of distributed tracing, _all_ components in your system should be instrumented.

</Tip>

## Tracing common configuration

Common tracing configuration contains global settings for all exporters.

### Service name

Set a service name for your router traces so you can easily locate them in external metrics dashboards.

The service name can be set by an environment variable or in [`router.yaml`](/router/configuration/overview#yaml-config-file), with the following order of precedence (first to last):

1. `OTEL_SERVICE_NAME` environment variable
2. `OTEL_RESOURCE_ATTRIBUTES` environment variable
3. `telemetry.exporters.tracing.common.service_name` in `router.yaml`

      <ExpansionPanel title="Example service_name">

      Example setting service name in `telemetry.exporters.tracing.common.service_name`:

      ```yaml title="router.yaml"
      telemetry:
        exporters:
          tracing:
            common:
              # (Optional) Set the service name to easily find metrics related to the apollo-router in your metrics dashboards
              service_name: "router" #highlight-line
      ```

      </ExpansionPanel>


4. `telemetry.exporters.tracing.common.resource` in `router.yaml`

      <ExpansionPanel title="Example resource">

      Example setting service name in `telemetry.exporters.tracing.common.resource`:

      ```yaml title="router.yaml"
      telemetry:
        exporters:
           tracing:
             common:
               resource:
                 # (Optional) Set the service name to easily find metrics related to the apollo-router in your metrics dashboards
                 "service.name": "router" #highlight-line
      ```

      </ExpansionPanel>

If the service name isn't explicitly set, it defaults to `unknown_service:router` or `unknown_service` if the executable name cannot be determined.

### `resource`

A resource attribute is a set of key-value pairs that provide additional information to an exporter. Application performance monitors (APM) may interpret and display resource information. 

In [`router.yaml`](/router/configuration/overview#yaml-config-file), resource attributes are set in `telemetry.exporters.tracing.common.resource`. For example:

```yaml title="router.yaml"
telemetry:
  exporters:
     tracing:
       common:
         resource:
           "deployment.environment.name": "production"
           "k8s.namespace.name": "{env.MY_K8_NAMESPACE_ENV_VARIABLE}"
```

For OpenTelemetry conventions for resources, see [Resource Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/README.md).

### `sampler`

You can configure the sampling rate of traces to match the rate of your application performance monitors (APM). To enable sampling configuration, in [`router.yaml`](/router/configuration/overview#yaml-config-file) set `telemetry.exporters.tracing.common.sampler` and `telemetry.exporters.tracing.common.parent_based_sampler`:

```yaml title="router.yaml"
telemetry:
  exporters:
     tracing:
       common:
         sampler: always_on # (default) all requests are sampled (always_on|always_off|<0.0-1.0>)
         parent_based_sampler: true # (default) If an incoming span has OpenTelemetry headers then the request will always be sampled. 
```

- `sampler` sets the sampling rate as a decimal percentage, `always_on`, or `always_off`.
  - For example, setting `sampler: 0.1` samples 10% of your requests.
  - `always_on` (the default) sends all spans to your APM.
  - `always_off` turns off sampling. No spans reach your APM.

- `parent_based_sampler` enables clients to make the sampling decision. This guarantees that a trace that starts at a client will also have spans at the router. You may wish to disable it (setting `parent_based_sampler: false`) if your router is exposed directly to the internet.

### `preview_datadog_agent_sampling`

<div className="flex flex-row items-start gap-2 mt-2">
  <MinVersionBadge version="Router v1.59" />
  <PreviewFeatureBadge />
</div>

Enable accurate Datadog APM views with the `preview_datadog_agent_sampling` option. 

The Datadog APM view relies on traces to generate metrics. For this to be accurate, all requests must be sampled and sent to the Datadog Agent.

To both enable accurate APM views and prevent _all_ traces from being sent to Datadog, you must set `preview_datadog_agent_sampling` to `true` and adjust the `sampler` to the desired percentage of traces to be sent to Datadog.

```yaml title="router.yaml"
telemetry:
  exporters:
    tracing:
      common:
        # Only 10 percent of spans will be forwarded from the Datadog agent to Datadog. Experiment to find a value that is good for you!
        sampler: 0.1
        preview_datadog_agent_sampling: true
```

To learn more details and limitations about this option, go to [`preview_datadog_agent_sampling`](/router/configuration/telemetry/exporters/tracing/datadog#preview_datadog_agent_sampling) in DataDog trace exporter docs.

### `propagation`

The `telemetry.exporters.tracing.propagation` section allows you to configure which propagators are active in addition to those automatically activated by using an exporter.

Specifying explicit propagation is generally only required if you're using an exporter that supports multiple trace ID formats, for example, OpenTelemetry Collector, Jaeger, or OpenTracing compatible exporters.

For example: 

```yaml title="router.yaml"
telemetry:
  exporters:
     tracing:
       propagation:
         # https://www.w3.org/TR/baggage/
         baggage: false
   
         # https://www.datadoghq.com/
         datadog: false
   
         # https://www.jaegertracing.io/ (compliant with opentracing)
         jaeger: false
   
         # https://www.w3.org/TR/trace-context/
         trace_context: false
   
         # https://zipkin.io/ (compliant with opentracing)
         zipkin: false
   
         # https://aws.amazon.com/xray/ (compliant with opentracing)
         aws_xray: false
   
         # If you have your own way to generate a trace id and you want to pass it via a custom request header
         request:
           # The name of the header to read the trace id from
           header_name: my-trace-id
           # The format of the trace when propagating to subgraphs.
           format: uuid
```

<Note>

You can't use Datadog propagation and other propagation types (except baggage) at the same time. Datadog uses a 64-bit trace ID format while other types use a 128-bit format, which causes conflicts in trace ID handling. If you attempt to enable both, the router will log a warning at startup. 

</Note>

#### `request` configuration reference

| Option        | Values                                                        | Default                           | Description                         |
|---------------|---------------------------------------------------------------|-----------------------------------|-------------------------------------|
| `header_name` |                                                               |  |  The name of the http header to use for propagation.                                   |
| `format`      | `hexadecimal`\|`open_telemetry`\|`decimal`\|`datadog`\|`uuid` | `hexadecimal`                     | The output format of the `trace_id` |

Valid values for `format`:
* `hexadecimal` - 32-character hexadecimal string (e.g. `0123456789abcdef0123456789abcdef`)
* `open_telemetry` - 32-character hexadecimal string (e.g. `0123456789abcdef0123456789abcdef`)
* `decimal` - 16-character decimal string (e.g. `1234567890123456`)
* `datadog` - 16-character decimal string (e.g. `1234567890123456`)
* `uuid` - 36-character UUID string (e.g. `01234567-89ab-cdef-0123-456789abcdef`)

<Note>

Incoming trace IDs must be in `open_telemetry` or `uuid` format.

</Note>

### Limits

You may set limits on spans to prevent sending too much data to your APM. For example:

```yaml title="router.yaml"
telemetry:
  exporters:
     tracing:
       common:
         max_attributes_per_event: 128
         max_attributes_per_link: 128
         max_attributes_per_span: 128
         max_events_per_span: 128
         max_links_per_span: 128
```

Attributes, events and links that exceed the limits are dropped silently.

#### `max_attributes_per_event` 

Events are used to describe something that happened in the context of a span. For example, an exception or a message sent. These events can have attributes that are key-value pairs that provide additional information to display via APM. 

#### `max_attributes_per_link`

Spans may link to other spans in the same or different trace. For example, a span may link to a parent span, or a span may link to a span in a different trace to represent that trace's parent. These links may have attributes that are key-value pairs that provide additional information to display via APM.

#### `max_attributes_per_span`

Spans are used to a activity in the context of a trace. For example, a request to a subgraph or a query planning. Spans can have attributes that are key-value pairs that provide additional information to display via APM.

#### `max_events_per_span`

Spans may have events that describe something that happened in the context of a span. For example, an exception or a message sent. The number of events per span can be limited to prevent spans becoming very large.

#### `max_links_per_span`

Spans may link to other spans in the same or different trace. For example, a span may link to a parent span, or a span may link to a span in a different trace to represent that trace's parent. The number of links per span can be limited to prevent spans becoming very large.

### `experimental_response_trace_id`

<ExperimentalFeatureBadge />

If you want to expose in response headers the generated trace ID or the one you provided using propagation headers you can use this configuration:

```yaml title="router.yaml"
telemetry:
  exporters:
     tracing:
       experimental_response_trace_id:
         enabled: true # default: false
         header_name: "my-trace-id" # default: "apollo-trace-id"
```
Using this configuration you will have a response header called `my-trace-id` containing the trace ID. It could help you to debug a specific query if you want to grep your log with this trace id to have more context.

#### `experimental_response_trace_id` reference

| Attribute      | Default           | Description                                           |
|----------------|-------------------|-------------------------------------------------------|
| `enabled`      | `false`           | Set to true to return trace IDs on response headers.  |
| `header_name`  | `apollo-trace-id` | The name of the header to respond with.               |



## Tracing common reference

| Attribute                        | Default                  | Description                                      |
|----------------------------------|--------------------------|--------------------------------------------------|
| `parent_based_sampler`           | `true`                   | Sampling decisions from upstream will be honored |
| `preview_datadog_agent_sampling` | `false`                  | Send all spans to the Datadog agent.             |
| `propagation`                    |                          | The propagation configuration.                   |
| `sampler`                        | `always_on`              | The sampling rate for traces.                    |
| `service_name`                   | `unknown_service:router` | The OpenTelemetry service name.                  |
| `service_namespace`              |                          | The OpenTelemetry namespace.                     |
| `resource`                       |                          | The OpenTelemetry resource to attach to traces.  |
| `experimental_response_trace_id` |                          | Return the trace ID in a response header.        |
| `max_attributes_per_event`       | 128                      | The maximum number of attributes per event.      |
| `max_attributes_per_link`        | 128                      | The maximum number of attributes per link.       |
| `max_attributes_per_span`        | 128                      | The maximum number of attributes per span.       |
| `max_events_per_span`            | 128                      | The maximum number of events per span.           |
| `max_links_per_span`             | 128                      | The maximum links per span.                      |

## Related topics

* [Connecting OpenTelemetry traces to Prometheus](/technotes/TN0003-opentelemetry-traces-to-prometheus)
* [Sending router traces and metrics to APM tools using OpenTelemetry](/graphos/reference/router/telemetry/metrics-exporters/overview)
